Skip to content

Disable tiered compilation in integration tests #3479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
andrewlock merged 2 commits into from
Nov 17, 2022
Merged

Disable tiered compilation in integration tests #3479

andrewlock merged 2 commits into from
Nov 17, 2022

Conversation

@andrewlock
Copy link
Member

@andrewlock andrewlock commented Nov 17, 2022

Summary of changes

Temporarily disable tiered compilation by setting COMPlus_TieredCompilation=0

Reason for change

We have been seeing flakiness in our integration tests for a long time which we have narrowed down to being related to how tiered JIT handles call-counting for our Rejited methods. This appears to be much more significant in our new Windows-2022 runners. We have raised an issue with the .NET runtime team. Disabling tiered compilation for now will hopefully significantly reduce flake in the meantime.

Implementation details

Set COMPlus_TieredCompilation=0 in integration sample apps. This is not a long-term solution, just a temporary fix for flake

Other details

For more details and analysis, see dotnet/runtime#77973

@andrewlock andrewlock added the area:tests unit tests, integration tests label Nov 17, 2022
@andrewlock andrewlock requested a review from a team as a code owner November 17, 2022 10:17
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 17, 2022

Datadog Report

Branch report: andrew/disable-tiered-jit
Commit report: 52cd15b

✅ dd-trace-dotnet 0 Failed, 0 New Flaky, 4 Passed, 0 Skipped, 32.39s Wall Time

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 17, 2022 •
edited
Loading

Datadog Report

Branch report: andrew/disable-tiered-jit
Commit report: 59fc775

❄️ dd-trace-dotnet 0 Failed, 1 New Flaky, 235465 Passed, 1285 Skipped, 19m 41s Wall Time

New Flaky Tests (1)

  • TestIastNotWeakRequest - Datadog.Trace.Security.IntegrationTests.Iast.AspNetCore5IastTests - Last Failure

    Expand for error 
     Results do not match.
 Differences:
 Received: Iast.NotWeak.AspNetCore5.IastEnabled.received.txt
 Verified: Iast.NotWeak.AspNetCore5.IastEnabled.verified.txt
 Received Content:
 [
   {
     TraceId: Id_1,
     SpanId: Id_2,
     Name: aspnet_core.request,
 ...

@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock requested review from a team as code owners November 17, 2022 11:18
@andrewlock
Copy link
Member Author

andrewlock commented Nov 17, 2022

Benchmarks Report 🐌

Benchmarks for #3479 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 776μs 1.09μs 4.09μs 0.383 0 0 3.22 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 556μs 298ns 1.15μs 0 0 0 2.63 KB
#3479 WriteAndFlushEnrichedTraces net472 775μs 996ns 3.86μs 0.381 0 0 3.22 KB
#3479 WriteAndFlushEnrichedTraces netcoreapp3.1 555μs 261ns 1.01μs 0 0 0 2.63 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 182ns 0.0377ns 0.141ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 244ns 0.569ns 2.2ns 0.00577 0 0 424 B
master AllCycleMoreComplexBody net472 180ns 0.0875ns 0.327ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 241ns 0.345ns 1.34ns 0.00551 0 0 400 B
master BodyExtractorSimpleBody net472 283ns 0.326ns 1.26ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 245ns 0.173ns 0.646ns 0.00369 0 0 272 B
master BodyExtractorMoreComplexBody net472 15.7μs 21.3ns 82.3ns 1.21 0.0156 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.8μs 6.1ns 23.6ns 0.0894 0 0 6.75 KB
#3479 AllCycleSimpleBody net472 182ns 0.0426ns 0.159ns 0.0676 0 0 425 B
#3479 AllCycleSimpleBody netcoreapp3.1 245ns 0.155ns 0.599ns 0.0058 0 0 424 B
#3479 AllCycleMoreComplexBody net472 181ns 0.036ns 0.13ns 0.0637 0 0 401 B
#3479 AllCycleMoreComplexBody netcoreapp3.1 237ns 0.142ns 0.552ns 0.00549 0 0 400 B
#3479 BodyExtractorSimpleBody net472 273ns 0.134ns 0.501ns 0.0574 0 0 361 B
#3479 BodyExtractorSimpleBody netcoreapp3.1 250ns 0.121ns 0.454ns 0.00364 0 0 272 B
#3479 BodyExtractorMoreComplexBody net472 15.3μs 13.7ns 51.4ns 1.2 0.0153 0 7.62 KB
#3479 BodyExtractorMoreComplexBody netcoreapp3.1 12.9μs 4.76ns 17.8ns 0.0907 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 176μs 234ns 906ns 0.263 0 0 20.44 KB
#3479 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3479 SendRequest netcoreapp3.1 174μs 194ns 750ns 0.26 0 0 20.44 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.77μs 1.14ns 4.43ns 0.16 0.000891 0 1.01 KB
master ExecuteNonQuery netcoreapp3.1 1.42μs 0.782ns 2.82ns 0.0135 0 0 1 KB
#3479 ExecuteNonQuery net472 1.74μs 0.777ns 3.01ns 0.16 0.000867 0 1.01 KB
#3479 ExecuteNonQuery netcoreapp3.1 1.4μs 0.569ns 2.2ns 0.0133 0 0 1 KB
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.51μs 1.27ns 4.92ns 0.193 0 0 1.22 KB
master CallElasticsearch netcoreapp3.1 1.5μs 0.373ns 1.35ns 0.0151 0 0 1.16 KB
master CallElasticsearchAsync net472 2.67μs 1.67ns 6.47ns 0.214 0 0 1.36 KB
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.55ns 2.13ns 0.0173 0 0 1.28 KB
#3479 CallElasticsearch net472 2.49μs 0.6ns 2.25ns 0.193 0 0 1.22 KB
#3479 CallElasticsearch netcoreapp3.1 1.51μs 0.686ns 2.66ns 0.0159 0 0 1.16 KB
#3479 CallElasticsearchAsync net472 2.62μs 1.29ns 4.84ns 0.215 0 0 1.36 KB
#3479 CallElasticsearchAsync netcoreapp3.1 1.63μs 0.53ns 1.91ns 0.017 0 0 1.28 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.69μs 0.855ns 3.31ns 0.236 0 0 1.49 KB
master ExecuteAsync netcoreapp3.1 1.73μs 1.64ns 6.35ns 0.0192 0 0 1.41 KB
#3479 ExecuteAsync net472 2.76μs 1.58ns 6.11ns 0.235 0 0 1.49 KB
#3479 ExecuteAsync netcoreapp3.1 1.76μs 0.581ns 2.25ns 0.0184 0 0 1.41 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.88μs 1.29ns 4.64ns 0.448 0 0 2.83 KB
master SendAsync netcoreapp3.1 3.67μs 1.33ns 5.15ns 0.0367 0 0 2.66 KB
#3479 SendAsync net472 5.85μs 2.06ns 7.99ns 0.449 0 0 2.83 KB
#3479 SendAsync netcoreapp3.1 3.56μs 3.17ns 11.9ns 0.0361 0 0 2.66 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.89μs 2.7ns 10.1ns 0.298 0 0 1.88 KB
master EnrichedLog netcoreapp3.1 2.39μs 1.12ns 4.36ns 0.0252 0 0 1.91 KB
#3479 EnrichedLog net472 2.89μs 2.61ns 10.1ns 0.297 0 0 1.88 KB
#3479 EnrichedLog netcoreapp3.1 2.38μs 0.802ns 3.11ns 0.026 0 0 1.91 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 150μs 91.2ns 341ns 0.675 0.225 0 4.72 KB
master EnrichedLog netcoreapp3.1 120μs 223ns 865ns 0 0 0 4.55 KB
#3479 EnrichedLog net472 149μs 93.2ns 361ns 0.745 0.223 0 4.72 KB
#3479 EnrichedLog netcoreapp3.1 118μs 126ns 489ns 0.059 0 0 4.55 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.56μs 1.65ns 6.4ns 0.579 0.00278 0 3.65 KB
master EnrichedLog netcoreapp3.1 4.24μs 2.27ns 8.49ns 0.0529 0 0 3.98 KB
#3479 EnrichedLog net472 5.49μs 1.87ns 6.99ns 0.58 0.00275 0 3.65 KB
#3479 EnrichedLog netcoreapp3.1 4.26μs 1.08ns 4.06ns 0.0534 0 0 3.98 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.28μs 0.772ns 2.89ns 0.228 0 0 1.44 KB
master SendReceive netcoreapp3.1 1.84μs 0.773ns 2.89ns 0.0192 0 0 1.38 KB
#3479 SendReceive net472 2.29μs 2.56ns 9.59ns 0.228 0 0 1.44 KB
#3479 SendReceive netcoreapp3.1 1.81μs 0.815ns 3.16ns 0.0191 0 0 1.38 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.8μs 1.54ns 5.95ns 0.364 0 0 2.3 KB
master EnrichedLog netcoreapp3.1 3.95μs 0.832ns 3ns 0.0256 0 0 1.86 KB
#3479 EnrichedLog net472 4.78μs 1.5ns 5.81ns 0.362 0 0 2.3 KB
#3479 EnrichedLog netcoreapp3.1 4.21μs 1.08ns 4.05ns 0.0249 0 0 1.86 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.15μs 0.386ns 1.49ns 0.139 0 0 875 B
master StartFinishSpan netcoreapp3.1 913ns 0.513ns 1.99ns 0.011 0 0 824 B
master StartFinishScope net472 1.32μs 0.295ns 1.06ns 0.152 0 0 955 B
master StartFinishScope netcoreapp3.1 1.07μs 0.308ns 1.19ns 0.0128 0 0 944 B
#3479 StartFinishSpan net472 1.19μs 1.02ns 3.93ns 0.139 0 0 875 B
#3479 StartFinishSpan netcoreapp3.1 969ns 0.686ns 2.66ns 0.0111 0 0 824 B
#3479 StartFinishScope net472 1.35μs 0.559ns 2.09ns 0.152 0 0 955 B
#3479 StartFinishScope netcoreapp3.1 1.08μs 0.487ns 1.89ns 0.0129 0 0 944 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.4μs 0.48ns 1.86ns 0.152 0 0 955 B
master RunOnMethodBegin netcoreapp3.1 1.14μs 0.327ns 1.13ns 0.0131 0 0 944 B
#3479 RunOnMethodBegin net472 1.45μs 0.389ns 1.51ns 0.151 0 0 955 B
#3479 RunOnMethodBegin netcoreapp3.1 1.17μs 0.488ns 1.83ns 0.0129 0 0 944 B

@andrewlock andrewlock merged commit 22990ad into master Nov 17, 2022
@andrewlock andrewlock deleted the andrew/disable-tiered-jit branch November 17, 2022 14:06
@github-actions github-actions bot added this to the vNext milestone Nov 17, 2022
@andrewlock
Copy link
Member Author

andrewlock commented Nov 17, 2022

Code Coverage Report 📊

✔️ Merging #3479 into master will not change line coverage
✔️ Merging #3479 into master will not change branch coverage
⛔ Merging #3479 into master will will increase complexity by 78

master #3479 Change
Lines 20547 / 27924 20523 / 27967
Lines % 74% 73% 0% ✔️
Branches 11890 / 17447 11925 / 17515
Branches % 68% 68% 0% ✔️
Complexity 18874 18952 78 ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3479 Change
Lines % 74% 73% 0% ✔️
Branches % 68% 68% 0% ✔️
Complexity 18874 18952 78 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.GitInfo -21% ⛔ -18% ⛔ 0 ✔️
Datadog.Trace.Propagators.SpanContextPropagator -8% ⛔ -3% ⚠️ 0 ✔️
Datadog.Trace.Propagators.DistributedContextExtractor -6% ⛔ -17% ⛔ 1 ⛔
Datadog.Trace.Debugger.PInvoke.DebuggerNativeMethods -6% ⛔ -10% ⛔ 0 ✔️

The following classes were added in #3479:

File Line coverage Branch coverage Complexity
Datadog.Trace.Iast.OverheadController 82% 100% 11
Datadog.Trace.Propagators.B3MultipleHeaderContextPropagator 100% 94% 33
Datadog.Trace.Propagators.SpanContextPropagatorFactory 100% 100% 21
Datadog.Trace.Propagators.W3CTraceContextPropagator 95% 93% 41

3 classes were removed from Datadog.Trace in #3479

View the full reports for further details:

@andrewlock andrewlock mentioned this pull request Nov 17, 2022
Merged
@tonyredondo tonyredondo mentioned this pull request Nov 24, 2022
Merged
@gleocadie gleocadie mentioned this pull request Nov 29, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tonyredondo tonyredondo tonyredondo approved these changes

@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:tests unit tests, integration tests

Projects

None yet

Milestone

2.20.0

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewlock @tonyredondo @pierotibou